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violated  at  singular  points  or  along  lines  in  the  image. 
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t.  Introduction 

Optical  How  is  the  distribution  of  apparent  velocities  of  movement  of  brightness  patterns  in  an  image. 
Optical  flow  can  arise  from  relative  motion  of  objects  and  the  viewer  (Gibson  1950.  1966).  Consequently, 
optical  flow  can  give  important  information  about  the  spatial  arrangement  of  the  objects  viewed  and  the 
rate  of  change  of  this  arrangement  (Gibson  1977).  Discontinuities  in  the  optical  flow  can  help  in  segment¬ 
ing  images  into  regions  that  correspond  to  dilferenl  objects  (Nak.iyama  &.  I  oomis  197-1)  Attempts  have 
been  made  lo  perform  such  segmentation  using  differences  between  succcshe  image  frame--  iJain  rt ,/!. 
1977,  Jain  eta!.  1979,  Jain  &  Nagel  1979.  I.inib&  Murphy  1975.  Nagel  1977).  Some  recent  papers  have 
considered  the  problem  of  recovering  the  motions  of  objects  relative  to  the  viewci  from  the  optical  flow 
(lladani  el  al.  1980,  Kocnderink  &  van  Doom  1975  &  1970,  I  onguet-l liggms  &  I’ra/dny  1979.  IVa/dny 
1979  &  1980).  In  some  cases  information  about  the  shape  of  an  object  may  also  be  recovered  ( Kocndermk 
&  van  Doom  1975  &  1976,  Clocksin  1978). 

These  papers  begin  by  assuming  that  the  optical  flow  has  already  been  determined  Although  some 
reference  has  been  made  to  schemes  for  computing  the  11  wv  from  successive  views  of  a  scene  (I'cnnema 
&  Thompson  1979.  lladani  el  al.  1980),  the  specifics  of  a  scheme  for  determining  the  flow  fiom  the  image 
have  not  been  described.  Related  work  has  been  done  in  an  attempt  to  formulate  a  model  for  the  short 
range  motion  detection  processes  in  human  vision  (Natali  &  I. ’liman  1980,  Marr<&  Ullman  1979). 

T  he  optical  flow  cannot  be  computed  at  a  point  in  the  image  independently  of  neighboring  points 
without  introducing  additional  constraints,  because  the  velocity  field  .it  each  image  point  has  two  com¬ 
ponents  while  the  change  in  image  brightness  at  a  point  in  the  image  plane  due  lo  motion  yields  only  one 
constraint.  Consider  for  example  a  patch  of  a  pattern  where  brightness  varies  as  a  function  of  one  image 
coordinate  but  not  the  other.  Movement  of  the  pattern  in  one  direction  alters  the  brightness  al  a  partic  ular 
point,  blit  motion  in  the  other  direction  yields  no  change.  Thus  components  of  movement  in  the  latter 
direction  cannot  he  computed  locally.  Additional  constraints  must  be  introduced  to  fully  determine  the 
flow. 


2.  Relationship  to  Object  Motion 

I  he  relationship  between  the  optical  flow  in  the  image  plane  and  the  velocities  of  objects  m  the  three 
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dimensional  world  is  not  necessarily  obvious.  We  perceive  motion  when  a  changing  picture  is  projected 
onto  a  stationary  screen,  for  example.  Conversely,  a  moving  object  may  give  rise  to  a  constant  brightness 
pattern.  Consider  for  example,  a  uniform  sphere  which  exhibits  shading  because  its  surface  elements  arc 
oriented  in  many  different  directions.  Yet,  when  it  is  rotated,  the  optical  flow  is  zero  at  all  points  in  the 
image,  since  the  shading  docs  not  move  with  the  surface.  Also,  specular  reflections  move  with  a  velocity 
characteristic  of  the  virtual  image,  not  the  surface  in  which  light  is  reflected. 


3.  The  Restricted  Problem  Domain 

For  convenience,  we  tackle  a  particularly  simple  world  where  die  apparent  velocity  of  brightness 
patterns  can  be  directly  identified  with  the  movement  of  surfaces  in  the  scene.  To  avoid  variations  in 
brightness  due  to  shading  effects  we  initially  assume  that  the  surface  is  flat.  We  further  assume  that 
die  incident  illumination  is  uniform  across  the  surface.  The  brightness  at  a  point  in  the  image  is  then 
proportional  to  the  reflectance  of  the  surface  at  the  corresponding  point  on  the  object.  Also,  we  assume 
initially  that  reflectance  varies  smoothly  and  has  no  spatial  discontinuities.  This  latter  condition  assures  us 
that  the  image  brightness  is  differentiable. 

In  this  simple  situation,  the  motion  of  the  brightness  patterns  in  the  image  is  directly  determined  by 
the  motions  of  corresponding  points  on  the  surface  of  the  object.  Computing  the  velocities  of  points  on 
the  object  is  a  matter  of  simple  geometry  once  the  optical  flow  is  known. 


4.  Constraints 


We  will  derive  an  equation  that  relates  the  change  in  image  brightness  at  a  point  to  the  motion  of  the 
brightness  pattern.  1  .et  the  image  brightness  at  the  point  ( x ,  y)  in  the  image  plane  at  time  t  be  denoted  by 
/’(/'  !/>  0  NJ<nv  consider  what  happens  when  the  pattern  moves.  The  brightness  of  a  particular  point  in 
the  pattern  is  constant,  so  that 
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l  sing  the  chain  rule  lor  dilferentiation  we  see  lh.U, 


dEdx  dE  dy  SE 
Ox  dt  "  chj  dt  ^  dt 


(See  Appendix  A  for  a  more  detailed  derivation.)  If  we  let 


dx  dy 

u=  n 


then  it  is  easy  to  see  that  we  have  a  single  linear  equation  in  the  two  unknowns  t<  and  u. 


E,u  d  EyV  |-  E,  -  0, 

where  we  have  also  introduced  the  additional  abbreviations  E,r  and  I',  for  the  partial  derivatives 
of  image  brightness  with  respect  to  x.  y  and  t.  respectively.  I  he  constraint  on  the  local  How  velocity 
expressed  by  this  equation  is  illustrated  in  figure  I.  Writing  the  equation  in  still  another  way, 

(£,,£*)•(«,«)  =  -Kt- 

thus  the  component  of  the  optical  llow  in  the  direction  of  the  brightness  gradient  [Et,  /•’,,)  equals 

E, 

We  cannot,  however  determine  the  component  of  movement  in  the  direction  of  the  iso  brightness  eon- 
toms,  at  right  angles  to  the  brightness  gradient.  As  a  consequence.  the  llow  velocity  (u,  v)  cannot  be 
computed  locally  w  ithout  introvlueingaddition.il  constraints. 


5.  Hie  Smoothness  (  oust  mint 

If  every  point  of  the  brightness  pattern  can  move  independently,  there  is  little  hope  of  recovering  the 
velocities.  More  commonly  we  v  iew  opaque  objects  of  finite  si/e  undergoing  rigid  motion  or  deformation. 
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Figure  1.  The  basic  rate  of  change  of  image  brightness  equation  constrains  the 
optical  flow  velocity.  The  velocity  (u,  u)  has  to  lie  along  a  certain  line  perpendicular 
to  the  brightness  gradient  vector  [Et,K„)  in  velocity  space. 
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In  this  case  neighboring  points  on  the  objects  have  similar  velocities  and  the  velocity  field  of  (he  bright¬ 
ness  patterns  in  the  image  varies  smoothly  almost  everywhere.  Discontinuities  in  flow  can  be  expected 
where  one  object  occludes  another. 

One  way  to  express  the  additional  constraint  is  to  limit  the  difference  between  the  flow  velocity  at 
a  point  and  the  average  velocity  over  a  small  neighborhood  containing  the  point,  l-quivalenlly  we  can 
minimize  the  sum  of  the  squares  of  the  1  aplaciansof  the  x-  and  (/-components  of  the  flow  .  The  I  aplucians 
of  u  and  v  are  defined  as 


V2u  = 


<9*ti  d^u 
Ox2  +  <v 


and 


VJv  = 


02v  dFv 
Bx1  By1 


In  simple  situations,  both  i  aplacians  are  zero.  If  die  viewer  translates  parallel  to  a  flat  object,  rotates  about 
a  line  perpendicular  to  the  surface  or  travels  orthogonally  to  the  surface  (assuming  perspective  projection), 
then  the  second  partial  derivatives  of  both  tt  and  v  vanish.  Note  that  our  approach  is  iti  contrast  w  ith 
that  of  I'cnnema  &  Thompson  (1079),  who  propose  an  algorithm  that  indirectly  incorporates  additional 
assumptions  such  as  surface  smoothness  or  object  rigidity. 


6.  Quantization  and  Noise 

Images  may  be  sampled  at  intervals  on  a  fixed  grid  of  points.  While  tesselations  other  than  the 
obvious  one  have  certain  advantages  (Merscrcau  1979,  Gray  1971),  for  convenience  we  will  assume  that 
the  image  is  sampled  on  a  square  grid  at  regular  intervals,  l  et  the  measured  brightness  be  at  the 
intersection  of  the  i-tli  row  and  jth  column  in  the  A-th  image  frame.  Ideally,  each  measurement  should 
be  an  average  over  the  area  of  a  picture  cell  and  ov  er  the  length  of  the  lime  internal.  In  the  experiments 
cited  here  we  have  taken  samples  at  discrete  points  in  space  and  time,  instead. 

In  addition  to  being  quantized  in  space  and  time,  the  measurements  will  in  practice  be  quantized  in 
brightness  as  well.  Further,  noise  will  be  apparent  in  measurements  obtained  in  any  real  system. 
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7.  Estimating  I  lie  Partial  Derivatives 

Wc  must  estimate  the  derivatives  of  brightness  from  the  discrete  set  of  image  brightness  measure¬ 
ments  available.  It  is  important  that  the  estimates  of  Es,  and  E,  be  consistent.  That  is,  they  should 
all  refer  to  the  same  point  in  the  image  at  the  same  time.  While  there  are  many  formulas  for  approximate 
differentiation  (Conte  &  de  Door  1965,  Hamming  1962)  we  will  use  a  set  which  gives  us  an  estimate  of  EJt 
Elr  Et  at  a  point  in  the  center  of  a  cube  formed  by  eight  measurements.  The  relationship  in  space  and 
time  between  these  measurements  is  shown  in  Figure  2.  Kach  of  the  estimates  is  the  average  of  four  first 
differences  taken  over  adjacent  measurements  in  the  cube. 

Es  |  fc  —  Eij,k 

+  i.e-H  —  Ejj.fc-H  4  ^f-t-tj-f  i,*+i  —  Ei+i,j,k' 4- 1} 

Ey  jM  ~  Ei.j.k  4  ^-t-I.J-t-l.fe  —Eij -t-l.fc 

4  Ei  +  t.j.fc+i  —  Ei,j,k+i  +Ei+ij+iik+i  —  Ei  j  H.fc-t-t} 

E,  +  l  Ei.j.k  4  t-t  —  Ei+w 

4  Ki.j+l,k+l  —  4^-t  IJ  +  l.fc  t  I  —  Ei  |  (J f  i,*}. 

Here  the  unit  of  length  is  the  grid  spacing  interval  in  each  image  frame  and  the  unit  of  time  is  the  image 

frame  sampling  period. 


8.  Estimating  the  Laplacian  of  the  Flow  Velocities 

Wc  also  must  approximate  the  l.aplacians  of  u  and  v.  One  convenient  approximation  takes  the 
following  form 

V*u  oa  K(u,  j  k  —  u.  j  t)  and  V4  pw  K(vl  j  k  —  u, t}  k), 
where  the  local  averages  ft  and  v  are  defined  as  follows 

hi.j.C  ~g{«i  \.J.C  4  W|,J  (  1.1-4  W,-t  lj.lt  4  W|.J  I, It} 

4  1 2 4  «i  ija  u  +  «i  i  i.j  i  i  t  4  »t  t  i.j  i*} 

V'j.c  ==g{,,i-  i.j.t  4  vi.j  tit  4  vt  fi.j.t  4  ty>~i,fc} 

4  j2 {wi  i i .it  4  w» —i, >~t-i.t  4  Vi  t \,j  \  i,/t  4  t  i.j  !,*}• 
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Figure  2.  The  three  partial  derivatives  of  image  brightness  at  the  center  of  the  cube 
are  each  estimated  from  the  average  of  first  differences  along  four  parallel  edges 
of  the  cube.  Here  the  column  index  j  corresponds  to  the  s  direction  in  the  image, 
the  row  index  i  to  the  y  direction,  while  k  lies  in  the  time  direction. 
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I'lto  proportionality  factor  k  equals  .1  if  the  average  is  computed  as  shown  and  assuming  again  (hat  (lie 
unit  oflengtli  equals  the  grid  spacing  interval,  figure  .1  illustrates  the  assignment  of  weights  to  neighbor¬ 
ing  points. 


9.  Minimization 

The  problem  then  is  to  minimize  tire  sum  of  the  errors  in  the  equation  for  tire  rate  of  change  of  image 
brightness. 

Sr,  =  Exu  -f  DyV  -f-  Kf, 

and  the  estimate  of  the  departure  from  smoothness  in  the  velocity  How, 


sc  =  (a~  u)2  +  0>  --  w)2- 


What  should  he  the  relative  weight  of  these  two  factors?  In  practice  the  image  brightness  measurements 
will  be  corrupted  In  quantization  error  and  noise  so  that  wc  cannot  expect  to  be  identically  zero.  This 
quantity  will  tend  to  have  an  error  magnitude  that  is  proportional  to  the  noise  in  the  measurement.  1'his 
fact  guides  us  in  choosing  a  suitable  weighting  factor,  denoted  by  a2.  as  will  be  seen  later. 

I  et  the  total  error  to  be  minimized  be 


S2  o262  -f  fi2. 

I  he  minimization  is  to  be  accomplished  by  finding  suitable  values  for  the  optical  How  velocity  (u,  u).  We 
dill'ercntiatc  82  to  obtain 

2«2(u  —  it)  2(Exu  |  Euv  f  E()Et 
2 al(i>  —  v)  f  2(E,u  f  EtJv  -f  E,)Ey. 

Setting  these  two  derivatives  equal  to  zero  leads  to  two  equations  in  it  and  v, 

(a2  -f-  C2)u  -f  EXE„ v  —  (a2tt  --  E, E,) 

EjEyii  -f  (o2  +  E'l)v  =  (a2v  -  EyE,). 


<962 

du 

OS2 

Ov 
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Figure  3.  The  Laplacian  is  estimated  by  subtracting  tire  value  at  a  poinl  from  a 
weighted  average  of  the  values  at  neighboring  points.  Shown  here  are  suitable 
weights  by  which  values  can  bo  multiplied. 
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The  determinant  of  the  coefficient  matrix  equals  o2(a2  -f-  E2  -j-  £2).  Solving  for  u  and  v  we  find  that 

(q2  +  I-'2X  +  E2u)u  =  -f  (a2  +  El)h  -  EtEuv  -  ExEt 
(a2  +  E]  -f-  E'l)v  =  -  Ej-Eyii  +  (a2  +  E2)u  -  EuEt. 


10.  Difference  of  Mow  at  a  Point  from  Local  Average 

These  equations  can  be  written  in  the  alternate  form 

(a2  -f  E]  +  El)(u  -  u)  =  -  E,\Eru  +  Eyv  +  Et) 

(a2  +  E2  +  El)(v  -  0)  =  -  Etf|E,u  +  Eu6  +  E(). 

This  shows  that  the  value  of  die  How  velocity  { u ,  v)  which  minimizes  the  error  82  lies  in  the  direction 
towards  the  constraint  line  along  a  line  that  intersects  the  constraint  line  at  right  angles.  This  relationship 
is  illustrated  geometrically  in  figure  4.  The  distance  from  the  local  average  is  proportional  to  the  error 
in  die  basic  formula  for  rate  of  change  of  brightness  when  u,  v  arc  substituted  for  u  and  v.  finally  we 
can  see  that  a2  plays  a  significant  role  only  for  areas  where  the  brightness  gradient  is  small,  preventing 
haphazard  adjustments  to  the  estimated  flow  velocity  occasioned  by  noise  in  the  estimated  derivatives. 
This  parameter  should  be  roughly  equal  to  the  expected  noise  in  the  estimate  of  E2  -j-  £2 . 


II.  Constrained  Minimization 

When  we  allow  a2  to  tend  to  zero  we  obtain  the  solution  to  a  constrained  minimization  problem. 
Applying  the  method  of  Lagrange  multipliers  (Russell  1976,  Yourgau  &  Mandelstam  1968)  to  the 
problem  of  minimizing  g2  while  maintaining  Sj,  =  0  leads  to 

(E2,  +  E2)(u  -**)  =  -  Er\E,u  +  E,,v  +  Ei] 

(El  +  E2)(tz  -t>)  =  -  Ey[E,u  +  E„ v  +  Et], 

We  will  not  use  these  equations  since  we  do  expect  errors  in  the  estimation  of  die  gradient  (Ex,  E,,). 
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Figu  re  4.  The  value  of  the  flow  velocity  which  minimizes  the  error  lies  on  a  line 
drawn  from  the  local  average  of  the  flow  velocity  perpendicular  to  the  constraint 
line. 


Horn  <f  Schmu  k 


I.  /.  t frnio  ' 


Page  12 


Determining  Ojnioil  How 


12.  Iterative  Solution 

We  now  have  a  pair  of  equations  for  each  point  in  the  image.  It  would  he  very  costly  to  solve  these 
equations  simultaneously  by  one  of  the  standard  methods,  such  as  tiauss-Jordan  elimination  (Hamming 
1%2,  I  lildchrand  1952).  The  corresponding  matrix  is  sparse  and  very  large  since  the  number  of  rows  and 
columns  equals  tw  ice  the  number  of  picture  cells  in  the  image.  Iterative  methods,  such  as  the  (iauss-Scidel 
method  (Hamming  1962,  Hildebrand  1956),  suggest  themselves.  We  can  compute  a  new  set  of  velocity 
estimates  (u"  t  1 ,  v”  *' 1 )  from  the  estimated  derivatives  and  the  average  of  the  previous  velocity  estimates 
(«",  v ")  by 

'  u"  "  =tin  -  +  E.v"  +  £,]/(o2  +  E 2  -f  Ey) 

v "  »  1  =t>"  -  Ey[Exun  +  Eyv'1  +  £(]/(a2  f-  E\  +  E*). 

It  is  interesting  that  the  new  estimates  at  a  particular  point  do  not  depend  directly  on  the  previous 
estimates  at  the  same  point. 


13.  hilling  In  Uniform  Regions 

In  parts  of  the  image  where  the  brightness  gradient  is  zero,  the  velocity  estimates  will  simply  be 
averages  of  the  neighboring  velocity  estimates.  There  is  no  local  information  to  constrain  the  apparent 
velocity  of  motion  of  the  brightness  pattern  in  these  areas.  Eventually  the  values  around  such  a  region  will 
propagate  inwards.  If  the  velocities  on  the  border  of  lire  region  arc  all  the  same,  then  points  in  the  region 
w ill  be  assigned  that  value  too  after  a  sufficient  number  of  iterations.  Velocity  information  is  thus  filled  in 
from  the  boundary  of  a  region  of  constant  brightness. 


14.  Number  of  Iterations 

If  the  values  on  the  border  arc  not  all  the  same,  the  values  filled  in  will  correspond  to  the  solution 
of  the  I  aplaee  equation  for  the  given  boundary  condition  (Antes  1977.  Milne  1953.  Richtmycr<&  Morion 
1957).  The  progress  of  this  filling  in  phenomena  is  similar  to  the  propagation  effects  in  the  solution  of  the 
heat  equation,  where  the  time  rate  of  change  of  temperature  is  proportional  to  ihc  I  aplacian.  This  gives 
us  a  means  of  understanding  the  iterative  method  in  physical  terms  and  of  estimating  the  number  of  steps 
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required.  I  ho  number  of  iior.it  ions  should  bo  larger  Ilian  the  cross  sec  linn  of  the  biggest  ic;..ion  '.but  luiist 
lv  Idled  in.  If  the  si/e  of  such  regions  is  not  known  in  advance  one  may  use  die  tn*s'-soi!ion  ><!  the  whole 
image  as  a  lonserv  alive  estimate. 


15.  Tightness  of  Constraint 

When  brightness  in  a  region  is  a  linear  function  of  the  image  coordinates  we  can  only  obtain  the 
component  of  optical  flow  in  the  direction  of  the  gradient.  The  component  at  right  angles  is  tilled  in  lion 
the  botmdais  of  the  region  as  described  hefoie.  In  general  the  solution  is  most  au  in  »(•  ! \  detettumed 
m  regions  uheie  the  In  iglilness  gradient  is  not  too  small  and  varies  in  direction  liom  point  to  is  u.t. 
Information  which  constrains  both  components  of  the  optical  How  velocity  is  then  available  in  a  (datively 
small  ncighboihood  ol  each  point.  Too  violent  Ihictuations  in  brightness  on  the  ode  i  hind  .ue  not 
dcsnable  since  the  estimates  of  derivatives  will  he  corrupted  due  to  uiulcis.mipting  .  aid  aliasing 


16.  Choice  of  Iterative  Scheme 

As  a  practical  m.ilier  one  has  a  choice  of  how  ihe  iterations  are  to  he  interlaced  vviili  the  time  steps 
On  the  one  hand,  one  could  iterate  until  the  solution  has  stabilized  before  advancing  to  the  next  mi  me 
frame.  On  the  other  hand,  given  a  good  initial  guess  one  may  need  only  one  delation  pei  time-sicp  A 
good  initial  guess  I  or  the  optica!  flow  velocities  is  usually  available  from  the  previous  lime  step. 

I  he  advantages  of  the  latter  approach  include  an  ability  to  deal  with  mote  image,  pci  unit  nine  md 
better  estimates  of  optica!  Dow  velocities  in  certain  regions  Areas  in  which  the  hi  iphliics-.  giadient  is  small 
lead  to  imeerl. lilt,  noisy  estimates  obtained  pailly  Ivy  tilling  in  from  ihe  sin  lomulmgs.  I  Iw  e  ninales  aic 
improved  In  coiisidcnug  I'm  (her  images.  Ihe  noise  in  measurements  of  the  images  will  lv  independent 
and  tend  to  cancel  out  Perhaps  more  mtpoil.mlh.  diDenmi  pails  of  the  p.ittei  n  w  ill  drill  In  a  given  point 
in  the  image 

A  practical  inipliiueiitaliuii  would  most  likely  employ  one  iteration  pin  lime  slep  I'm  these  reasons 
We  ilhisli.itc  both  appio.icln"  in  the  experiments. 
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17.  Kxpcriiucnts 

I  he  iterative  scheme  has  been  implemented  and  applied  to  image  sequences  corresponding  to  a 
number  of  simple  How  patterns.  The  results  shown  here  are  on  a  relatively  small  image  of  .12  by  .12 
picture  cells.  The  brightness  measurements  were  intentionally  corrupted  by  approximately  1%  noise  and 
then  qu.mli/ed  into  256  levels  to  simulate  a  real  imaging  situation.  The  underlying  surface  reflectance 
pattern  was  a  linear  combination  of  spatially  orthogonal  sinusoids.  Their  wavelength  was  chosen  to  gave 
reasonably  strong  brightness  gradients  without  leading  to  uudersnmplmg  problems.  Discontinuities  were 
avoided  to  ensure  that  the  required  derivatives  exist  everywhere. 

Shown  in  Figure  5  for  example  are  four  frames  of  a  sequence  of  images  depicting  a  sphere  rotating 
about  an  axis  inclined  towards  the  viewer.  A  smoothly  varying  reflectance  pattern  is  painted  on  the  surface 
of  the  sphere.  The  sphere  is  illuminated  uniformly  from  all  directions  so  that  there  is  no  shading. 


18.  Results 

The  lirst  flow  to  be  investigated  was  a  simple  linear  translation  of  the  entire  brightness  pattern.  I  lie 
resulting  computed  flow  is  shown  as  a  needle  diagram  in  Figure  6  for  1.  4,  16.  and  64  iterations.  The 
estimated  flow  velocities  are  depicted  as  short  lines,  showing  die  apparent  displacement  during  one  time 
step.  In  (Ins  example  a  single  lime  step  was  taken  so  that  the  computations  aie  based  on  just  two  images. 
Initially  the  estimates  of  flow  velocity  are  zero.  Consequently  the  first  iteration  shows  vectors  in  the 
direction  of  the  brightness  gradient,  l  ater,  the  estimates  approach  the  correct  values  in  all  parts  of  the 
image  Few  changes  occur  after  .12  iterations  when  the  velocity  vectors  have  errors  of  about  10%.  I'he 
estimates  tend  to  be  too  small,  lather  than  too  large,  perhaps  because  of  a  tendency  to  underestimate  the 
derivatives,  l  he  worst  errors  occur,  as  one  might  expect,  where  the  brightness  gradient  is  small. 

In  the  second  experiment  one  iteration  was  used  per  time  step  on  the  same  linear  translation  image 
sequence.  I  he  resulting  computed  flow  is  shown  in  Figure  7  for  I.  4,  16,  and  64  time  steps.  I'he  estimates 
appro.iv h  the  correct  values  more  rapidly  and  do  not  have  a  tendency  to  be  too  small,  as  in  the  previous 
experiment,  l  ew  changes  occur  after  16  iterations  when  the  velocity  vectors  have  errors  of  about  7%.  I'he 
worst  errors  occur,  as  one  might  expect,  where  the  noise  in  recent  measurements  of  brightness  was  worst. 


t.  /.  Memo  5/2 


Dorn  d  Schmick 


tfii  ms 


Firjiire  h  I  'our  fiaruos  mil  of  a  sequence  of  imayes  of  a  sphere  rotation  about  an 
axis  inclined  towards  thi;  viewer.  Tire  sphere  is  covered  with  a  pattern  which 
varies  smoothly  from  place  to  place.  The  sphere  is  poitrayed  .lyamst  a  fixed,  liciltt I y 
textured  flacky round.  Iiirayc  sequences  like  tlrese  ar«r  processed  tiy  (lie  optical  How 
algorithm. 
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Figure  6.  Flow  pattern  computed  tor  simple  translation  of  a  brightness  pattern.  The 
estimates  after  1,  4,  16,  and  64  iterations  are  shown.  The  velocity  is  0.5  picture 
cells  in  the  z  direction  and  1.0  picture  cells  in  the  y  direction  per  time  interval. 
Two  images  are  used  as  input,  depicting  the  situation  at  two  times  separated  by 
one  time  interval. 
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IPS  .illur  1  4,  Hi.  und  64  lime  slops  are  shown.  Hero  one  iler.itmn  is  used 
;„;r  „n'o  Slop.  Convergence  is  more  rapid  and  the  ve.out.es  are  estimated  more 

accurately. 
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While  individual  estimates  of  velocity  may  not  be  very  accurate,  the  average  over  the  whole  image  was 
within  1%  of  the  correct  value. 

Next,  the  method  was  applied  to  simple  rotation  and  simple  contraction  of  live  brightness  pattern. 
The  results  after  32  time  steps  .ire  shown  in  figure  8.  Note  that  the  magnitude  of  the  velocity  is  propor¬ 
tional  to  the  distance  from  the  origin  of  the  llow  in  boil,  these  cases. 

In  all  of  the  examples  so  far  the  I  aplacian  of  both  llow  velocity  components  is  zero  everywhere.  We 
also  studied  more  difficult  cases  where  this  was  not  the  case.  In  particular,  if  we  let  the  magnitude  of  the 
velocity  vary  as  the  inverse  of  the  distance  from  the  origin  we  generate  llow  around  a  line  vortex  and  two 
dimensional  llow  into  a  sink.  The  computed  (low  patterns  are  shown  in  Figure  9.  Here  the  compulation 
involved  many  iterations  based  on  a  single  time  step.  The  worst  errors  occur  near  the  singularity  at  the 
origin  of  the  llow  pattern,  where  velocities  arc  found  which  arc  much  larger  than  one  picture  cell  per  time 
step. 

Finally  we  considered  rigid  body  motions.  Shown  in  Figure  10  arc  the  flows  computed  for  a  cylinder 
rotating  about  its  axis  and  for  a  rotating  sphere.  In  both  cases  the  laplacian  of  the  llow  is  not  zero  and  in 
fact  the  I  aplacian  of  one  of  the  velocity  components  becomes  infinite  on  the  occluding  bound.  Since  the 
velocities  themselves  remain  finite,  reasonable  solutions  are  still  obtained.  The  correct  llow  patterns  are 
shown  in  Figure  II.  Comparing  the  computed  and  exact  values  shows  that  the  worst  errors  occur  on  the 
occluding  boundary.  These  boundaries  constitute  a  one  dimensional  subset  of  the  plane  and  so  one  can 
expect  that  the  relative  number  of  points  at  which  the  estimated  flow  is  seriously  in  error  will  decrease  as 
the  resolution  of  the  image  is  made  finer. 

In  Appendix  B  it  is  shown  that  there  is  direct  relationship  between  the  I  .aplacian  of  the  flow  velocity 
components  and  the  I. aplacian  of  the  surface  height.  This  can  be  used  to  sec  how  our  smoothness  con¬ 
straint  will  lair  for  different  objects.  For  example,  a  rotating  polyhedron  will  give  rise  to  a  flow  which  has 
zeio  I  aplacian  except  on  the  image  lines  which  are  die  projections  of  die  edges  of  die  body. 

19,  Suimutiry 

A  method  was  developed  for  computing  optical  flow  from  a  sequence  of  images.  It  is  based  on  the 
observation  that  the  flow  velocity  has  two  components  and  that  the  basic  equation  for  the  rate  of  change 
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Figure  0.  Flow  patterns  computed  for  simple  rotation  and  simple  contraction  ol  a 
brightness  pattern.  In  t ho  lust  case,  the  pattern  is  rotated  about  2.8  degrees  per 
time  step,  while  it  is  contracted  about  b"o  per  time  step  in  ttie  second  case,  the 
estimates  altei  32  time  steps  are  shown. 
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Figure  9.  Flow  patterns  computed  for  flow  around  a  line  vortex  and  two  dimensional 
flow  into  a  sink.  In  each  case  the  estimates  after  32  iterations  are  shown. 
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Figure  10.  Flow  patterns  computed  for  a  cylinder  rotating  about  its  axis  and  for  a 
rotating  sphere.  The  axis  of  the  cylinder  is  inclined  30  degrees  towards  the  viewer 
and  that  of  the  sphere  45  degrees.  Both  are  rotating  at  about  5  degrees  per  time 
step.  The  estimates  shown  are  obtained  after  32  time  steps. 
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of  image  brightness  provides  only  one  constraint.  Smoothness  of  the  llow  was  intiodoced  as  a  second 
constraint.  An  iterative  method  for  solving  the  resulting  ct|iiniion  was  then  developed.  A  simple  im¬ 
plementation  prov  ideil  visual  conlirmation  of  convergence  of  the  solution  in  the  form  ol  needle  diagiams. 
Txamples  of  several  dilferent  types  of  optical  (low  patterns  were  studied.  These  included  cases  where  the 
I  aplacian  of  the  llovv  was  zero  and  cases  where  it  became  infinite  at  singular  points  or  along  bounding 
curves. 
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Appendix  A.  Rule  of  Change  of  image  Brightness 


Consider  a  patch  of  the  brightness  pattern  that  is  displaced  a  distance  6 x  in  die  x-dircction  and  6y  in 
the  y-dircction  in  time  6t.  The  brightness  of  the  patch  is  assumed  to  remain  constant  so  that 


E{x,  y,  t)  =  E(x  -f  Sx,  y  Sy,  t  -|-  61). 


Expanding  the  right  hand  side  about  the  point  (x,  y,  £)  we  get, 

E(x,  y,  t)  -  E(x,  y,  t)  +  6x g  +  fiy~  +  6t~  +  e. 

Where  e  contains  second  and  higher  order  terms  in  6x,  6y,  and  St.  After  subtracting  E(x,  y,  t)  front  both 
sides  and  dividing  through  by  6t  we  have 


SxdE  .  6ydE  .  dE  .  A 


where  0(fi£)  is  a  term  of  order  61,  and  we  assume  that  fix  and  fiy  vary  as  6t.  In  the  limit  as  61  0  this 

becomes 


<9 E  dx  dE  dy  dE 

didi^dydt  +  dt 
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Appendix  B.  Smoothness  of  Mow  for  Rigid  Body  Motions 


I  el  a  rigid  body  rotate  about  an  axis  (car,  u»,y>  u>.),  where  the  magnitude  of  die  vector  equals  the 
angular  velocity  of  the  motion.  If  this  axis  passes  through  the  origin,  then  the  velocity  of  a  point  lx,  y,  z) 
equals  the  cross  product  of  (wx,  ceu,  w=),  and  (x,  y,  z).  There  is  a  direct  relationship  between  the  image 
coordinates  and  the  x  and  y  coordinates  here  if  we  assume  that  the  image  is  generated  by  orthographic 
projection.  The  x  and  y  components  of  the  velocity  can  be  written, 

u  =ui,jZ  — 

V  —U)2X  —  UJjZ . 


Consequently, 

V2ti  =  +  OJyV^Z 

Vlv  =  —  u>xV2z. 

This  illustrates  that  the  smoothness  of  die  optical  flow  is  related  directly  to  the  smoothness  of  die  rotadng 
body  and  that  the  Laplacian  of  die  flow  velocity  will  become  infinite  on  the  occluding  bound,  since  the 
partial  derivatives  of  z  with  respect  to  x  and  y  become  infinite  there. 
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